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1. Introduction 


The fruition of university courses has significantly changed in the last decade, in consequence 
of the higher accessibility of technological devices: universities, as well as for-profit companies, 
started to propose video lectures, as a substitution or in support of traditional lessons, and massive 
online open courses (MOOCs) have gained more and more importance in the education processes. 
This tendency comes as an answer to a growing need for flexibility expressed by working 
students, life-long learning processes, students who have families and care burdens, those with 
some forms of disability or special needs that make it difficult to attend classes. 

Many authors have investigated the effectiveness of video lectures, primarily in 
comparison with face-to-face classes, with mixing results: some found no significant 
differences between online and face-to-face courses (Lim et al., 2007; Neuhauser, 2002; 
Nemetz et al., 2017), while others suggested higher outcomes in online courses (Soffer and 
Nachmias, 2018; Burkhardt et al., 2008; Connolly et al., 2007; Lim et al., 2008). The Covid- 
19 pandemic has magnified and accelerated the surge of online teaching, in a way that makes the 
change hardly reversible. The ongoing debate on the effectiveness of video lectures in higher 
education is meant to last and intensify. 

In this paper, we describe and discuss the implementation, the acceptability, and the 
effectiveness of an experimental service designed to capture, record, edit and stream video 
lectures; this system was introduced with the principal aim of supporting, and not substituting, 
in-class learning. In detail, Section 2 illustrates the experimental service and the main usage 
behaviours; Section3 presents the main results in terms of effectiveness and usage models, 
while some conclusions are drawn in Section 4. 


2. ONELab: an experimental education support 


ONELab is a system designed to capture, record, edit and stream video lectures, 
introduces by the Department of Communication and Economics of the University of Modena 
and Reggio Emilia in September 2017. Traditional face-to-face classes were regularly held, 
but ONELab was intended to ease the educational experience of those students who cannot 
attend classes regularly, and to provide an additional support to traditional students. Each 
classroom is equipped with a video camera pointed on the teacher’s desk, an audio system to 
capture and amplify the teacher’s voice, a screen to display the slideshow, and a live video 
production system to capture, mix, record and stream the video signals (i.e. teacher’s video and 
slideshow) and the audio. After a minimal post-processing, the video lectures are loaded to 
the online platform and made available for students (see Furini et al., 2018; 2020 for more 
details). 

In the first year of experimentation, from September 2017 to June 2018, 1,376 video 
lectures were produced, covering the 49 courses offered in the first year of the five bachelor’s 
and master’s degrees supplied by the Department, for a total of 2,064 hours. In the academic 
year 2018/19 these numbers doubled, and further increased in 2019/20, as the courses offered 
in the second and third year joined the experimentation. 
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The analysis of the first year log files shows that students’ reaction was enthusiastic, with 
an average of 8,323 video lectures played each month in the first year, from September 2017 
to August 2018, and a peak of 14,483 views in January 2018, during the exams session. 
Overall, during the year students watched video lectures for 71,488 hours. The most watched 
lectures relate to technical subjects, such as mathematics and statistics, where students take 
advantage from the possibility to replay some difficult passages until they are clear. 

Video lectures are watched mostly during the teaching semester, but a significant part of 
students resort to watch them when the semester is over, especially during the exams sessions. 
The usage analyses show that students watch video lessons mostly during the working hours, 
from Monday to Friday; however, 16% of the views happen during the week-ends, and 22% 
in the evening and during the night, suggesting that, when given the opportunity, students 
tend to customise the learning process to their needs and life-style. 

Of the 1251 freshmen in the academic year 2017/18, only 319 (25.4%) never accessed the 
ONELab platform to watch video lessons during their first year, while 13.4% never accessed 
it, neither in the first or in the second year. Table 1 shows the percentage of non-users among 
different categories of students, separately for undergraduate and graduate students. 


Table 1. Percentage of students who never accessed the ONELab video lectures during the 
first year and in the first two years, per students’ characteristics. 


Graduate Undergraduate Total 
First year | Two years | First year | Two years | First year | Two years 
Males 8.4 1.1 30.4*** | 23.8*** | 25,5*** 18.7*** 
Females 3.8 2.2 16.1*** 13.1*** 12.3*** Q FERF 
SLD 0.0 0.0 0.0*** 0.0*** 0.0*** 0.0*** 
Non-SLD 5.4 1.8 22.9% *%* 18.2*** 18.8*** 13.7*** 
Italian 4.9° e 22 Ast 18.0** ISAE 13.2*** 
EU 0.0 0.0** 0.0*** 0.0** 0.0*** 0.0*** 
Non-EU 22.2° 22.28% A I tata 20.0** 25.6*** | 20.3*** 
Lyceum 5.5 1.2 18.6° 14.4° 13.8** 9.6** 
Technical college 3.3 2.6 27.1° 217° 22.6** 17.8** 
Vocational college 4.1 0.0 23.0 20.0 19.4 16.1 
Other school dA 7.1 17.9 12.5 15.7 11.4 
Dropouts 18.5° 14.8* AN3A | AA OFF | ARIF | AL 2AE 
Non-dropouts 4.0° 0.4* 13.4*** 6.9% #* 10.4*** 4.8% 7% 
Total 5.4 1.8 22.6 17.9 17.9 13.4 


Significance level: *** 99.9%, ** 99%; * 95%; ° 90%. 


The use of ONELab is particularly popular among graduate students, while almost one 
undergraduate student out of four never watched any video lecture. Females are more 
conscientious than males, and look for every provided support to enhance their preparation, 
but the difference is statistically significant only among undergraduates. Students affected by 
Specific Learning Disorders are only 12, but none of them missed the new learning support, 
that allows for a certain degree of self-paced study, ensuring more control over their learning. 
On the other hand, undergraduate students coming from technical and vocational colleges are 
less organised in their study, and overlook video lectures to a greater extent, while non-EU 
foreigners miss this support both as undergraduate and graduate students. 

The recourse to video lectures is extremely scarce among students who end up dropping 
out the university in the first year. To some extent, these students might have dropped out 
because they did not take advantage of ONELab to support their studies, but it might also be 
that some students decided to leave the university, or to transfer to a different degree, so early 
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that they did not have time to try the video lectures. 


3. Effectiveness of the video lectures 


The high percentage of usage is a first indirect indicator of effectiveness of the video lectures, 
but we aim at assessing the benefits of the video lectures in terms of learning outcomes, namely 
the number of acquired (European) credits and the final grades. We focus on students enrolled in 
2017, and analyse data on their ONELab accesses and academic achievements during the 
academic years 2017/18 and 2018/19. We do not consider the third year, neither for three-year 
courses, because it corresponds to the Covid-19 pandemic outbreak, when face-to-face classes 
were totally replaced by video lectures in the second semester, which makes the situation 
incomparable. 

Since early dropouts produce a low number of credits (or no credits at all) and a low level of 
access to the online platform, we remove them from the following analyses to exclude the 
existence of spurious relationships. 

Table 2 shows that the average number of credits acquired by students who watched video 
lectures is largely bigger than for those who did not, both in the first and in the second year, and 
the difference is statistically significant. Students who accessed ONELab also performed better in 
terms of grades: the average grade of accessing students is higher than for non-accessing students, 
although the difference is only marginally significant in the second year. However, separate 
analyses carried out on undergraduate and master students demonstrate that students who 
accessed the video lectures show significantly better performances only in terms of acquired 
credits. 


Table 2. Average number of acquired credits during the first year and average grade for ONELab 
users and non-users, per degree level. 


A First year Second year 
Credits Average grade Credits Average grade 
= „| ONELab users 36.9 23.5 
3 g| No ONELab users 17.0 23.4 
D E Total 33.4 23.5 
Z ol T test 7.51 0.20 
> S| (p-value) (p < 0.001)| (p = 0.841) 
ONELab users 38.2 26.3 
o No ONELab users 26.5 25.6 
S E| Total 37.6 26.3 
S S| Test 2.07 0.99 
O 2| (p-value) (p = 0.039)| (p = 0.323) 
o ONELab users 37.3 24.5 
S | No ONELab users 18.0 23.8 
S | Total 34.6 24.4 
— | Ttest 8.38 1.957 
< | (p-value) (p < 0.001) | (p= 0.051) 


The rough distinction between students who never watched video lectures and those who 


accessed the platform at least one time, although simplistic, has proven to be meaningful in 
explaining performance differences among students. We try to describe in more detail the 
different usage styles of those who accessed the platform at least one time in the two years 

through the following variables, separately measured on the first and the second year: 
= Total number of accesses: this variable measures the general degree of usage during the 
first year. It varies between 0 and 864 for the first year, and between 0 and 1,885 in the 
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second year; the average is respectively 75.3 and 112.7; 

= Total number of different courses accessed: this provides information on the students’ 
choice of using the platform for one, some, or all of the scheduled courses. It varies 
between 0 and 25 in the first year, and between 0 and 28 in the second, and the average is 
4.8 and 4.7 in the two occasions; 

= Number of courses accessed at least 5 times: the wide range of observed values registered 
for the previous index suggests that some students have browsed in the platform clicking 
on more courses than the ones planned for their study course; this variable filters the 
courses that are only accessed on a random browse, and measures the number of courses 
accessed at least 5 times; both in the first and in the second year it varies between 0 and 
13, and the average is 3; 

= Number of courses accessed at least 10 times: this variable measures the number of 
courses accessed at least 10 times, signalling a larger commitment to the course; it varies 
between 0 and 9 in the first year, and between 0 and 11 in the second, and the average is 
2.1 and 2.2; 

= Maximum number of accesses to a single course: this index shows how many times each 
student played the video-lectures of the course he accessed most in the year; for the first 
year it ranges from 0 to 253, and the average is 27.3, while for the second year it ranges 
between 0 and 698 and the average is 44.1. 


Some of these variables show unexpectedly high values (for example, students registering 
1,885 total accesses, or students who played 698 times the video lectures of a single course), and 
the reason is twofold. First, every single access does not correspond to a complete play of the 
video-lecture; as reported by many students, “critical” passages, especially on some technical 
topics, have been repeatedly reloaded and re-played, and sometimes a lecture is erroneously 
played while looking for another one, or for a different part of the same recording. In addition, the 
platform was a novelty that probably raised curiosity among students, leading some to explore the 
resources far beyond the actual usage. 

Based on the ten described variables, we perform an agglomerative hierarchical cluster 
analysis; the agglomeration criterion is the Ward’s method that, at each step, merges the couple of 
units/clusters that leads to minimum increase in total within-cluster variance. The distance is the 
squared Euclidean. Given the different order of magnitude, all variables have been rescaled to the 
[0-1] range using min-max normalization. 

This cluster analysis suggests the existence of four distinct groups; combining these clusters 
with the group of absolute non-users, we obtain the five profiles described in Table 3 (first year 
dropouts are excluded from the analysis): 

1. Absolute non-users: They are 9% of all students; they never accessed the ONELab 

platform in the two academic years. 

2. Episodic users: This group amounts to the 32.9% of all students; on average, they tried to 
play a few video-lectures from about 3 courses in the first year and only a couple on the 
second, played a single course about 5-10 times and almost never accessed more than 10 
times to a single course. 

3. Regular users: They represent 25% of students; on average, they accessed most of the 
courses planned in their study program, but chose 2-3 of them which were played more 
frequently, up to 40 times. In the second year their usage intensity declines, and they play 
less videos, from less courses, a smaller number of times; the experience during the first 
year helps them distinguish which courses are worth watching and re-watching and which 
are not. 

4. Converted users: They are 23.6% of all students; during the first year, they show a scarce 
recourse to video lectures, larger than episodic users but far from regular users. 
Nevertheless, during the second year their usage intensity grows and exceeds regular 
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users. They are probably students who discovered the video lectures only late in the first 
year, or approached them first without enthusiasm but found out they were useful than 
expected for their preparation. 

5. Zealous users: They amount to 9.5% of students, and they accessed the platform hundreds 
of times; they accessed more or less all of the courses provided in their study program, 
and they were assiduous on most of them, playing video-lectures from each single course 
up to 70 times in the first year, and even up to 120 in the second year. 


Table 3. Average number of accesses, average number of courses accessed, average number of 
courses accessed at least 5 times, average number of courses accessed at least 10 times, and 
maximum number of accesses to a single course, for each group 


Accesses Courses Courses Courses Maximum n° 
accessed accessed 5 accessed 10 of accesses to 
times or more | times or more | a single course 
yearl | year2 | yearl | year2 | yearl | year2 | yearl | year2 | yearl | year2 
Absolute 0 0 0 0 0 0 0 0 0 0 
non-users 
Episodic 20.7 16.3 3.1 1.9 1.2 0.8 0.6 0.4 11.2 10.5 
users 
Regular 135.3 | 93.2 7.6 5.8 5.8 3.5 4.1 2.4 46.3 42.9 
users 
Converted | 47.6 | 190.3 | 4.9 7.4 2.6 4.7 1.4 3.6 22.4 | 77.9 
users 
Zealous 260.9 | 413.9 8.3 9.8 6.9 78 5.9 6.7 74.0 | 121.7 
users 
Total 76.6 | 112.7 | 4.9 4.7 3.1 3.0 2.1 2.2 27.5 | 44.1 


For each group of students, the learning performances are reported in Table 4. The level of 
performance increases with the frequency of usage of the ONELab services. Regarding the 
number of credits, all the group means are statistically different at least at a 95% significance 
level, except for Converted users, that are not significantly different from Regular users in the first 
year, and from Zealous users in the second year. The average grade shows only slight differences, 
nevertheless consistent with a better performance for regular and zealous users. Differences 
between graduate and undergraduate students are not noticeable. 


Table 4. Average number of acquired credits and average grade, in the first and second year, for 
each group, per degree level. 


First year Second year 
Credits Average grade Credits Average grade 
Non-users 9.6 23.4 
Episodic users 24.0 23.8 
Regular users 36.7 25.3 
Converted users 49.2 24.2 
Zealous users 53.8 24.6 
Total 34.6 24.4 


4. Conclusions 


In this paper, we analysed the effectiveness of an experimental platform to provide university 
students with remote access to video lectures to support traditional face-to-face classes. Results 
show higher learning outcomes for students who regularly watched the video lectures, primarily 
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in terms of the number of acquired credits. This is consistent with the conclusions drawn in 
Cagliero et al. (2017), who report higher student’s success rates following the introduction of an 
analogous system to provide video-recorded lessons to complement in-class learning. In our 
experience, the beneficial is particularly pronounced for undergraduate students, although they 
show a more limited recourse to the platform than graduate students. 

However, a more careful analysis of the principal beneficiaries of the implemented service 
casts a shadow on the capacity of the system to smooth learning ability differences and recover 
those students who have a hard time keeping pace with their studies and exams. Video lectures, in 
fact, are mainly watched by conscientious students, i.e. females, students coming from “lyceum” 
high school, and graduate students, who aim at improving their learning through additional 
educational material, while critical students are those who access the platform less. This suggests 
that the information about the new service should be conveyed to students in a more careful and 
focused way, addressing especially to students at risk of being left behind and dropping out. In 
this sense, given the strong connection between dropouts and video lectures (non) usage, 
monitoring and analysing the access data might help to detect critical students, and try to prevent 
them from dropping out. 

Finally, a negative consequence of the introduction of this service was a dramatic decrease in 
the number of students attending classes, much before the university classrooms were emptied by 
the pandemic crisis. When face-to-face classes will return to normality after more than one year of 
online teaching, the problem is likely to become even more compelling, forcing teachers and 
pedagogues to rethink face-to-face classes in a more interactive and engaging format. 
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